One-Shot Talking Face Generation from Single-Speaker Audio-Visual Correlation Learning

Audio-driven one-shot talking face generation methods are usually trained on video resources of various persons. However, their created videos often suffer unnatural mouth shapes and asynchronous lips because those struggle to learn a consistent speech style from different speakers. We observe that it would be much easier specific speaker, which leads authentic movements. Hence, we propose novel framework by exploring correlations between audio visual motions speaker then transferring audio-driven motion fields reference image. Specifically, develop an Audio-Visual Correlation Transformer (AVCT) aims infer represented keypoint based dense input audio. In particular, considering may come identities in deployment, incorporate phonemes represent signals. this manner, our AVCT can inherently generalize spoken other identities. Moreover, as keypoints used speakers, is agnostic against appearances the training thus allows us manipulate images readily. Considering lead motions, field transfer module exploited reduce gap identity reference. Once obtained image, employ image renderer generate its clip. Thanks learned speaking style, method generates vivid Extensive experiments demonstrate synthesized outperform state-of-the-art terms quality lip-sync.